Quantum Phase Transition in the SU(4) Spin-Orbital Model on the Triangular Lattice 
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Motivated by the absence of cooperative Jahn- Teller effect in LiNi02 and BaVS3, two layered 
oxides with triangular planes, we study the SU(4) symmetric spin-orbital model on the triangular 
lattice. Upon reducing the next-nearest neighbour coupling, we show that the system undergoes a 
quantum phase transition to a liquid phase. A variational approach to this liquid phase shows that 
simple types of long-range correlations are suppressed, suggesting that it is stable against lattice 
distortions. 



The possibility to enhance quantum fluctuations and 
destroy magnetic long-range order by orbital fluctuations 
was first proposed a few years ago in the pioneering work 
of Feiner et al Q . A careful analysis of the spin-orbital 
model of cubic perovskites with two-fold orbital degener- 
acy revealed the presence of spin liquid phases in which 
orbital degrees of freedom form valence-bond patterns. 
The search for experimental realizations of this intrigu- 
ing physics has been going on since then, but the best 
candidate remains the old system LiNiC>2 0, a layered 
S=l/2 system which does not undergo any phase tran- 
sition down to the lowest temperatures. In this system, 
Ni 3+ ions are arranged in triangular layers, and since the 
SU(2) Heisenberg model on a triangular lattice is now 
believed to sustain long-range order || , the origin of this 
behaviour has logically been attributed to the two-fold 
orbital degeneracy of Ni 3+ in the low spin state. How- 
ever, the absence of any structural distortions, which one 
would expect e.g. from a valence bond-ordered spin liq- 
uid of Ref. [Q , is still a mystery. Similar physics seems to 
be present in BaVS3 B, a 3d 1 system with stacked tri- 
angular planes: Between 30K and 70K it develops a spin 
gapped phase, again without significant lattice distor- 
tions. We believe that essential aspects of this interme- 
diate phase are also describable in terms of a triangular 
spin-orbital model with twofold orbital degeneracy. 

On the theory side, it has become clear since the work 
of Feiner et al [jlj that states in which orbitals form 
valence-bond dimer patterns do not exhaust the possi- 
bilities of spin liquid states. In particular, an increas- 
ing attention has been devoted to the SU(4) symmetric 
Kugel-Khomskii model [pi, where the orbital and spin 
degrees of freedom play symmetric roles. In terms of 
spin-1/2 operators Oi for the spins and pseudo-spin 1/2 
operators fi for the orbitals, this model is defined by the 
Hamiltonian: 
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(id) 



This Hamiltonian can be rewritten as Ti = J^V ~ JijPij, 
where Pi a permutes the states of the four-dimensional 



fundamental representations of SU(4) at sites i and j. In 
ID, this model has a gapless spectrum with a fourfold 
periodicity of correlation functions B . The investigation 
of this model on other simple lattices has revealed that, 
whenever possible, there is a tendency to form four-site 
plaquettes to accommodate SU(4) singlets 0, in contrast 
to the orbital valence bond solids. These plaquettes have 
been unambiguously shown to form a well defined pat- 
tern in the case of ladders [||, and evidence that this 
might also be the case for the square lattice has been 
put forward [[)[ |l(J. Alternatively, long range order has 
been proposed in Ref. jll). It is interesting to note that 
this SU(4) model is also a good starting point to describe 
CeB 6 (e.g. Ref. while the other version with conju- 

gate representations on different sublattices is more ap- 
propriate in the context of strong coupling to the lattice 
|l3| , or to perform a 1/N expansion |]l4j| . 

In this Letter, we present strong evidence, all based on 
arguments that respect the underlying SU(4) symmetry, 
that the SU(4) model on a triangular lattice with nearest- 
neighbour (NN) exchange J > only is also a spin liquid, 
but unlike previous suggestions (?], 10 , we put forward 
detailed variational arguments that go against the pres- 
ence of plaquette order in the ground state. Extending 
the model by including a coupling J' > between next- 
nearest neighbours (NNN), we prove that the model un- 
dergoes a quantum phase transition between a disordered 
state for J' = and an ordered state for large enough J'. 
The emerging picture of the disordered phase is that of 
a true liquid both in spin and orbital sector, with ac- 
cordingly no tendency to undergo any distortion when 
coupled to the lattice. 

Spin-waves. -The infinite degeneracy of the antiferro- 
magnetic, four-state Potts model on the triangular lattice 
is lifted by J' > 0, leading to a unique, four-sublattice 
ordering pattern (see Fig. 0(a)). Once the classical state 
is ordered, quantum fluctuations can be analyzed within 
a flavor- wave theory p5[ , which is a generalization of 
the standard SU(2) linearized spin-wave theory. First, 
one extends the fundamental representation on a sin- 
gle site into a Young diagram with a single row and M 
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FIG. 1: (a) Four-sublattice long-range ordered state with ba- 
sis vectors ui = (1,0) and 112 = (— 1/2, v3/2). The dashed 
line represents the next-nearest neighbor exchange. The sub- 
lattices are labelled according to the occupation by the states 
of the fundamental representation, (b) Brillouin-zone of the 
triangular lattice. The smaller, shaded hexagon denotes the 
reduced Brillouin-zone associated with the 4-sublattice order. 



columns (the fully symmetric representation). The SU(4) 
operators can be conveniently expressed by a general- 
ized Holstein-PrimakofT transformation, where the fluctu- 
ations on each site are described by three different bosons 
b™(l) with to 7^ n, for which the vacuum state is the Af- 
ford direct product of the state |m) associated with the 
particular sublattice (Fig. 0(a)). Following the standard 
procedure p5| , i.e. expanding in 1/M and keeping the 
quadratic bosonic terms, the Hamiltonian can be diago- 
nalized and, up to a constant, is given by: 
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The energies of the flavour modes are given by 



oj mn (k) = 2^(J + J') 2 - [Jcos(kr mn ) + J' cos(kr^„J] 2 

(3) 

where r m „ and r' mn are the NN and NNN distances be- 
tween sublattices to and n. There are 12 branches in the 
reduced Brillouin zone (RBZ, Fig. 0(b)), which is equiv- 
alent to 3 in the normal Brillouin zone. The operators a 
and b are related through a Bogoljubov transformation. 

In Fig. H we show a plot of the flavour dispersion 
for some arbitrary value of J'/ J. For J' = 0, the en- 




FIG. 2: Flavor- wave dispersion for J' /J — 0.1. Inset: Re- 
duced moment as a function of J' /J. 



ergy of the flavour excitations becomes effectively one- 
dimensional Q (e.g. wi2(k) = 2J|sinfca;|, with zero 
energy node along the k x = line). These low energy 
quantum fluctuations destroy the 4-sublattice long-range 
order. Including J', the magnons along the Y — M line 
acquire finite dispersion cx V J J', and this will reduce the 
fluctuations and stabilize the LRO state. Quantitatively 
the fluctuations are reflected in the effective staggered 
moment (M) = M - (£^ m b ?\ l ) b ?i l ))i which reads 



(M) = M 



E 



j + j' 



-I) 
00 2/ RBZ 



(4) 



The logarithmic divergence is the manifestation of the 
low energy modes. The (M) for M = 1 is finite when 
J'/ J > 0.12 (inset in Fig. ^), and we take this criterium 
for the stability of the LRO. Note that, within the linear 
flavor wave theory, the staggered moments and the en- 
ergy are symmetric in J and J'. So we expect LRO to 
be stable for 0.12 < J' /J < 8 (for J = we have four 
decoupled intercalated triangular lattices). 

Tower of states - To assess the validity of these results 
in the extreme quantum limit of the fundamental repre- 
sentation, we have looked for signatures of the quantum 
phase transition in the exact spectrum on a finite system. 
Specifically, we have numerically diagonalized a 12-site 
cluster defined by the lattice vectors Ti = 2ui + 4u2 
and T2 = 4ui + 2u2. This particular cluster inherits 
the Dq symmetry of the triangular lattice, and it is also 
compatible with the 4-sublattice ordering. Our aim was 
to identify the existence of the so-called Anderson tower 
spectrum, a set of states well separated from the others, 
which become degenerate and give rise to a linear combi- 
nation with LRO in the thermodynamic limit j^, |l6| . For 
SU(2) models, their energy for an iV-site system, mea- 
sured from the ground state, is proportional to S 2 /iV, 
where the total spin S is the addition of sublattice spins 
which maximize the sublattice magnetizations |p7| . In 
the SU(4) case, the same formula is expected to hold 
if S 2 is replaced by the first Casimir operator C of the 
SU(4) algebra (see Ref. || for definition and convention). 

These states can be easily identified in Fig. [|(a) , which 
shows the spectrum for J' /J — 0.2. In particular, the de- 
generacies and the symmetry properties are in complete 
agreement with group theoretical calculations |18|. This 
structure is also present in the spectra obtained for for 
all values of J'/ J larger than 0.2 up to J' = J, including 
,/' = J/2, a special point for the 12-site cluster where 
the tower structure can be explicitly proven |l^]. By 
contrast, the spectrum for J' = is shown in Fig. ||(b). 
Clearly, the tower structure is lost: The excitation ener- 
gies are not aligned on a straight line, and the symmetry 
and degeneracy of the corresponding excited states do 
not agree with group theory. This behaviour is consis- 
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FIG. 3: Low-energy spectrum of the 12-site cluster as a func- 
tion of the first Casimir C for J'/J = 0.2 (a) and J' = 
(b). The Anderson tower of states is clearly identified for 
J'/J — 0.2 (upper plot). The Young-diagrams corresponding 
to the states that build up the tower are depicted. 



tent with a quantum phase transition between J'/J = 
and 0.2. Due to the very rapid increase of the size of the 
Hilbert space, we could not study larger systems to check 
that the tower structure is still there for large enough 
J'/J, and to perform a finite-size scaling of the slope. 
Still, the comparison of the spin-wave results, obtained 
in the thermodynamic limit, with these exact diagonal- 
ization results gives a very strong indication that there is 
a quantum phase transition between a four-sublattice or- 
dered state for large enough J'/J and some kind of spin 
liquid for J' = 0. 

Variational approach - What is the nature of this dis- 
ordered ground state? For SU(2) models, experience 
with frustrated systems has shown that it is useful to 
look at the structure of the low-lying singlet spectrum 
of a finite system, which can contain a single state in 
valence bond solids with no lattice-symmetry breaking, 
four states for true spin liquids on a torus (topological 
degeneracy JTif), a finite number of states in valence 
bond solids with lattice-symmetry breaking, or a pro- 




FIG. 4: Three remarkable covering patterns of the triangular 
lattice with plaquettes. The last two resonate strongly. 
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TABLE I: Energy, degeneracy, and wave- vector of the low- 
lying singlet states of the 12 and 16 site symmetric clusters 
obtained for J = 1 and J' — by exact diagonalization and 
by the variational approach with 0, 1 and 2 additional steps. 
The second column is the energy of a single covering (no res- 
onance between plaquettes). The number of plaquette cover- 
ings is equal to 36 (84) for 12 (16) sites, while there are 462 
(24024) singlets in the full Hilbert space. In the exact diago- 
nalization spectrum, these states lie below the first multiplet 
(E = -12.841 and -18.451 for 12 and 16 sites respectively). 



liferation of low-lying singlets, as in the spin-1/2 kagome 
antiferromagnct pc[ . In the present case, exact diagonal- 
ization results on finite-size clusters with 12 and 16 sites 
show that there are 4 (resp. 6) singlets before the first 
multiplet, but no clear tendency can be identified. 

To reach larger system sizes, we have decided to re- 
sort to variational calculations in the spirit of the RVB 
approach to frustrated SU(2) magnets pi] ]. There are 
significant differences though. First, one needs at least 
four sites to make an SU(4) singlet. Now, among all 
possible 4-site clusters, the particular role of the singlet 
plaquettes was pointed out by Li et al. 0: On a tri- 
angular lattice, each plaquette has a diagonal energy of 
— (13/4) J, which is much lower than the diagonal energy 
of the LRO state (which is 0). Thus the singlet plaquettes 
are a good starting point for a variational calculation. To 
implement the method, we define a Hilbert space which is 
spanned by the different coverings \ipj) (j = 1, . . . , N cov ) 
of the finite size cluster with singlet plaquettes. The 
eigenenergies are then solution of the generalized eigen- 
value problem det (EOij — -By) = 0, where Oy = (i/>i\ipj) 
and Hij = {ipi\TL\ipj) . A systematic way to improve the 
method is to double the Hilbert space by including the 
states = H\ipj), which leads to the 
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2N cov x 2N cov eigenvalue problem, where — 
(ipi\H n \ipj) . This step can be repeated to include ma- 
trix elements of further powers of Ti n . The efficiency of 
the method is illustrated in Table. |[ 

The advantage of the variational approach is that we 
could perform calculations for much larger clusters, up 
to 36 sites with 2 additional steps, 48 sites with 1 ad- 
ditional step, and 64 sites with no additional step. For 
clarity, we concentrate the discussion on two natural pos- 
sibilities: i) The singlet plaquettes form a pattern with 
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FIG. 5: Dominant plaquette covering in the ground state of 
the 48 site cluster (dashed line). The diamonds making up 
the new superlattice are shaded using different grays for the 
two patterns of Fig. ^|(b). 



long-range order, the most natural candidate being then 
the static covering (plaquette array) as in the square lat- 
tice H (See Fig. f|(a)); ii) The plaquettes undergo strong 
resonances between configurations that differ only locally 
the smallest pattern that allows such a resonance being 
the diamond depicted in Fig. f|(b). 

The first possibility is definitely not favoured by our 
variational results: For both the 36-site and the 48-sitc 
clusters, the plaquette covering which has the largest 
weight in the ground state wave function is not the square 
covering, but a diamond covering (see Fig. |sj> . This fact 
is actually easy to understand: The matrix elements cor- 
responding to the resonance of Fig. |](b) dominate the 
secular equation, and they are completely suppressed in 
the square pattern of Fig. [|(a). 

The effect of these resonances on the ground state cor- 
relations is a delicate issue. The ground states of the 
36-site and 48-site clusters are consistent with a long- 
range pattern for the diamond covering of the triangular 
lattice, but given the very small sizes from that point of 
view (the 36 and 48 site cluster can accommodate 3 and 4 
diamonds respectively) this result is probably not signifi- 
cant. What is likely to be more relevant is the amplitude 
of the processes that would enter a description in terms 
of an effective Hamiltonian. The two plaquette coverings 
inside a diamond can be described by an fsing variable, 
and the resonance gives rise to a transverse field of order 
J/2. The spectrum obtained for the 36-site and 48-site 
clusters are then consistent with a very small potential 
energy term with two and three body interactions. A de- 
tailed analysis of this model lies beyond the scope of the 
present paper, but given the prominence of the kinetic 
energy term, it is very likely that the system is in the 
equivalent of the disordered phase of the Ising model in 
a transverse field p2| . 

In summary this variational approach strongly sug- 
gests that the ground state is a plaquette liquid with no 
four-site plaquette long-range order, and with strong lo- 
cal resonances between the configurations of Fig. ^(b). 
The presence of long-range order associated to a specific 
pattern of diamond covering can neither be ascertained 
nor excluded on the basis of the present results. We just 



note that, if no longer range correlations are present, this 
plaquette liquid is expected, as its SU(2) counterparts, to 
exhibit topological degeneracy |lS| ]. 

Conclusion - To conclude, we have shown that it is 
possible to order the SU(4) Heisenberg model for suffi- 
ciently large nearest neighbor repulsion, without the need 
to introduce anisotropy, and we have identified a quan- 
tum phase transition around J' /J = 0.12. For J' = 0, 
which corresponds to a minimal model of LiNi02, we 
have shown that the ground state is a spin and orbital 
liquid, and we have shown that simple objects like dimers 
of plaquettes do not develop long-range order. These re- 
sults are consistent with the absence of any kind of phase 
transition in LiNiC>2. More generally, Mott insulators 
with orbital degeneracy and the appropriate geometry 
emerge as potential candidates for completely order-free 
spin liquids with topological degeneracy. 
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